APPARATUS AND METHOD FOR AUTHORING MULTIMEDIA CONTENTS 
WITH OBJECT-BASED INTERACTIVITY 



Field of the Invention 

The present invention relates to an apparatus and method 
for authoring multimedia contents with object-based 
interactivity; and, more particularly, to an object-based 
interactive multimedia contents authoring apparatus and method 
capable of authoring multimedia contents, which include media 
objects such as video, audio, static images, graphics, 
animation, character information, etc. and user interactivity 
of an object unit, on an object basis and re-editing the 
contents on the object basis, and a computer readable 
recording medium on which a program for implementing the 
method is recorded. 



Background of the Invention 



In general, video, audio and static image 
editing/authoring tools consider each of video, audio and 
static images as an independent object and provide the object 
with nonlinear editing functions for cutting and attaching the 
object at an arbitrary position and special effect functions, 
e.g. , fade in/out. 

However, in case of utilizing editing/authoring methods 
provided by the conventional editing/authoring tools, since 



each of the video, audio and static images is considered as 
the independent object, it is impossible to edit each of 
bodies existing in the video, audio and static images and to 
execute user interactive editing capable of showing various 
5 contents according to users' demands while the contents are 
displayed . 

Further, although one content can be authored by using 
video, static images and graphic objects during editing the 
video, an edited/authored resulting product is generated as 
10 one video and, therefore, it is almost impossible to re-edit 
™ the edited/authored resulting product. 

That is, since conventional moving image and static 
image authoring apparatuses only provide methods which 
classify the video, audio and static images into independent 
15 objects, and edit and author the independent objects, they 
l-v fail to provide object-based multimedia contents authoring 
!?'>■ methods which treat each body existing in the video and static 
images as one object and, further, methods for editing user 
interactivity . 

20 

Summary of the Invention 

It is, therefore, a primary object of the present 
invention to provide an object-based interactive multimedia 
25 contents authoring apparatus and method for treating each body 
existing in video, audio and static images as one object, 
thereby providing various editing/authoring functions for 
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object-based interactive multimedia contents, easily re- 
editing edited/authored results, and executing interactive 
manipulation on an object basis to thereby edit user 
interactivity through which it is possible to respond to 
5 various demands of users, and a computer readable recording 
medium on which a program for implementing the method is 
recorded. 

In accordance with one aspect of the present invention, 
there is provided an apparatus for authoring multimedia 
I)§) contents with object-based interactivity, which comprises: 
yjj a user interfacing unit for providing an interface to 

fi'j thereby edit object-based interactive multimedia contents by 
; using a multimedia information editing and authoring tool; 
i, 4 an editorial information processing unit for converting 

the multimedia contents supplied from the user interfacing 
Jj! unit on an object basis to the form applicable to an object- 
H * based internal material structure supporting the editorial 
information authoring, storing the converted contents, and 
changing the form of the interactive multimedia contents 
2 0 information stored as the internal material structure to the 
file form so as to perform an input or output process of the 
contents; and 

a media coding and decoding unit for encoding and 
decoding the interactive multimedia contents information 
25 provided from the editorial information processing unit. 

In accordance with another aspect of the present 
invention, there is provided an object-based interactive 
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multimedia contents authoring method for use in an object- 
based interactive multimedia contents authoring apparatus, 
comprising the steps of: 

securing a new internal material structure and a new 
5 authoring space on a user interface, and receiving a plurality 
of parameters or initializing the authoring space to preset 
defaults ; 

authoring object-based interactive multimedia contents 
by inserting and deleting media objects based on the 
1.0 initialized authoring space and editing the user interactivity 
\pi on an object basis and properties of objects; and 
flj storing the authored object-based interactive multimedia 

fjj contents in a binary or text form. 

|k In accordance with still another aspect of the present 

3i§ invention, there is provided a computer readable medium on 
i| which a program used in implementing an object-based 
interactive multimedia contents authoring apparatus employing 
a processor is recorded, comprising: 

a first function for securing a new internal material 
20 structure and a new authoring space on a user interface, and 
receiving a plurality of parameters or initializing the 
authoring space to preset defaults; 

a second function for authoring object-based interactive 
multimedia contents by inserting and deleting media objects 
25 based on the initialized authoring space and editing the user 
interactivity on an object basis and properties of objects; 
and 
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a third function for storing the authored object-based 
interactive multimedia contents in a binary or text form. 

Brief Description of the Drawings 

5 

The above and other objects and features of the 
present invention will become apparent from the following 
description of preferred embodiments given in conjunction with 
the accompanying drawings, in which: 
J 6 Fig. 1 shows a block diagram of an object-based 

interactive multimedia contents authoring apparatus in 
111 accordance with an embodiment of the present invention; 
|M Fig. 2 illustrates a detailed block diagram of an 

M editorial information processing unit in Fig. 1; 
l|jt Fig. 3 is a detailed block diagram of an object 

j;. editorial information processor in Fig. 2; 

Fig. 4 provides a detailed block diagram of a media 
coding/decoding unit in Fig. 1; and 

Fig. 5 represents a flow chart of showing an object- 
2 0 based interactive multimedia contents authoring method in 
accordance with an embodiment of the present invention. 

Detailed Description of the Preferred Embodiments 

25 Hereinafter, with reference to the drawings, a preferred 

embodiment of the present invention will be explained in 
detail . 
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Referring to Fig. 1, there is illustrated a block 
diagram of an object-based interactive multimedia contents 
authoring apparatus in accordance with an embodiment of the 
present invention. 
5 As shown in Fig. 1, the inventive interactive multimedia 

contents authoring apparatus comprises a user interfacing unit 
103 for providing an interface through which users can easily 
manipulate editing/authoring tools, an editorial information 
processing unit 104 for editing contents and user inputs and a 
|0 media coding/decoding unit 105 for coding and decoding 

edited/authored multimedia contents. 
J|t The user interfacing unit 103 analyzes and processes the 

^ user inputs provided through an input device 101, provides the 
* ! editorial information processing unit 104 with editorial 
1|| information, and displays edited/authored results through an 
Ijj output device 102. 

I s * Herein, the user interfacing unit 103 includes an 

interface for inserting and deleting each object and editing 
properties of characterizing each object, an interface for 

2 0 editing logical relationships between media objects, an 
interface for editing the time allocation such as a display 
starting time and a display ending time of each object, an 
interface for editing the spatial allocation determining a 
position of each media object in contents, an interface for 

2 5 editing user interactivity and an interface for showing 
information for the contents. 

Each of the interfaces consists of an interface having a 
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graphic user interface (GUI) form capable of simply editing 
properties of media objects by using a mouse and an interface 
for editing exact values by using a keyboard. Herein, the 
properties characterizing each media object represent 
5 characteristic properties of the media object, e.g., a shape 
of the media object, and the logical relationship between 
media objects is a property representing grouping information, 
a dependent relationship between media objects and so on. 

The user interactivity means that the users can freely 
iap manipulate positions, display starting times and display 
ending time of the media objects during displaying the 
|J{ edited/authored interactive multimedia contents. At this time, 
% f{ the users can select, watch or listen to contents what they 
* want and, according to requests of the users, it is possible 
f§ to execute scene change, scene update, scene insertion or 
IaI scene removal. 

I** The editorial information processing unit 104 performs a 

function for storing the user inputs provided from the user 
interfacing unit 103 and the information for the media objects 

2 0 used in the contents editing/authoring in an internal material 
structure and preserving editing/authoring information for 
stored contents as files having a text type and a binary type 
and a function for reading in the editing/authoring 
information for the contents stored as the files of the text 

2 5 type and the binary type and storing the editing/authoring 
information in the internal material structure to make the 
information re-editable. 
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The media coding/decoding unit 105 implements a media 
coding/decoding technique following an international standard 
or a non-standardized coding/decoding technique so as to use 
various types of media objects including video objects of 
5 arbitrary shapes in the contents editing/authoring. 

Herein, it is preferable that the media coding/decoding 
unit 105 includes hardware dedicated for performing media 
processing, thereby executing the media coding/decoding 
f «. function in real-time or a higher speed than real-time. 
|3? Referring to Fig. 2, there is shown a detailed block 

|J; diagram of the editorial information processing unit 104 in 
!,!! Fig. 1 in accordance with an embodiment of the present 
^ invention. 

Jt &s illustrated in Fig. 2, the editorial information 

|1 processing unit 104 includes a data access application program 
fcl interface (API) 201 in charge of exchanging information with 
the user interfacing unit 103, an object editorial information 
processor 2 02 for converting editorial information delivered 
from the outside to the form of being applicable to the 
2 0 internal material structure and then storing the converted 
editorial information as internal materials, an object-based 
internal material structure 203 for storing the 
editing/authoring information of the object-based interactive 
multimedia contents therein and a file input/output processor 
25 204 for performing data inputting and outputting 106 and 107 
of the edited/authored results related to a storage and 
executing a form conversion between the internal material 
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structure and the input/output files. 

The data access API 2 01 restricts the access from the 
outside to the internal material structure 203 to be 
accomplished therethrough. 
5 The object-based internal material structure 2 03 reads in 

the contents stored in the storage, stores them as internal 
materials, and preserves the editing/authoring information 
provided from the outside as the internal materials so as to 
edit /author current contents. Here, in performing the 

IjO contents editing/ authoring process, there may be some users 
who want to author what they are aiming at by themselves in a 
^ higher level and assign a lower level process to 
j"; editing/authoring tools. On the other hand, there may exist 
* users who want to edit even details of the contents, e.g., 
IS frames in video editing, by themselves. That is, some users 
III may want to edit the contents in an expert level. 
I** Therefore, according to the purpose in use, the object- 

based internal material structure 203 can have an internal 
material structure supporting the lower level authoring for 
20 the general public who does not have expert knowledge about 
the object-based interactive multimedia contents authoring, an 
internal material structure supporting the higher level 
authoring for experts, or both of the above two internal 
material structures. 
25 As an example of the internal material structure for the 

higher level authoring, there is 0-profile of XMT (extensible 
MPEG-4 Textual format; ISO/IEC 14496-1 Amd3 ) of MPEG-4 systems. 
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On the other hand, as the internal material structure for the 
lower level authoring, there is an internal material structure 
supporting A-profile of XMT. 

The file input/output processor 204 includes a file 
analyzing module 205 for reading in the object-based 
interactive multimedia contents stored in the storage and 
storing the contents in the object-based internal material 
structure 203 and a file generating module 206 for 
transferring edited/authored results of the object-based 
interactive multimedia contents stored in the object-based 
internal material structure 2 03 to the storage. 

The file input/output processor 204 further includes a 
form converting module so as to perform form conversion 
between the internal materials and the input/output files. 

When being stored in the storage, the interactive 
multimedia contents being edited/authored resulting products 
can be stored in a text form noticeable by people to thereby 
effectively represent an intention of an author and perform 
easy re-editing of the contents. Furthermore, the interactive 
multimedia contents can be stored in a binary form so as to 
reduce an amount of data to be stored, accomplish easy 
distribution to final users of the contents and perform 
streaming. Therefore, it is preferable that the file 
input/output processor 204 supports both of the text file form 
and the binary file form. 

The file analyzing module 2 05 executes a function for 
reading in the contents of the text form stored in the storage 
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and storing them in the object-based internal material 
structure 2 03, a function for reading in the contents of the 
binary form stored in the storage and storing them in the 
object-based internal material structure 203, or reading in 
5 both of the contents of the text form and the contents of the 
binary form and storing them in the object-based internal 
material structure 2 03. Meanwhile, as occasion demands, the 
file analyzing module 205 can have a function for reading in 
the contents stored in the storage and examining logical 
W errors and errors on the meaning of the contents. 
\p The file generating module 2 06 performs a function for 

01 preserving the contents stored in the object-based internal 
P material structure 203 in the storage in the text form or the 
binary form, or a function for storing the contents in the 
lflj storage in the text form and the binary form. The form 
lij converting module converts the results of the higher level 
l, A authoring to those of the lower level authoring in case of the 
editing/authoring tool providing both of the higher level 
editing and the lower level editing and changes the 
20 edited/authored contents to a file form of a higher level 
which is not supported by the editing/authoring tool. 

Referring to Fig. 3, there is provided a detailed block 
diagram of the object editorial information processor 2 02 in 
Fig. 2 in accordance with an embodiment of the present 
25 invention. 

The object editorial information processor 202 executes a 
function for examining whether the editorial information 
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supplied from the outside is proper or not, a function for 
generating, deleting or changing the internal material 
structure for each media object to store the editorial 
information as the internal material structure and a function 
5 for converting the information stored in the internal material 
structure to the form that the outside can refer to. 

As described in Fig. 3, the object editorial information 
processor 202 includes a time allocation editorial information 
processing module 301 for processing the editorial information 
10 for the time allocation of each media object, a spatial 
% allocation editorial information processing module 302 for 
^ processing the editorial information for the spatial 
allocation of each media object, a user interactivity 
f editorial information processing module 303 for processing the 
f| editorial information for the user interactivity and a 
fij property and logical structure editorial information 
fs.<* processing module 304 for processing the editorial information 
for properties characterizing each media object, e.g., a shape 
of each media object. 
2 0 The object editorial information processor 2 02 further 

includes an object description information processing module 
for converting object description information stored in the 
internal material structure to the form that the outside can 
refer to and the object description information processing 
25 module uses MPEG-7 based object description. 

The time allocation editorial information processing 
module 301 examines whether time allocation editorial 
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information supplied from the outside is proper or not in 
order to edit the time allocation information such as a 
display starting time and a display ending time of each media 
object existing in the contents when the object-based 
5 interactive multimedia contents being the edited/authored 
resulting products are displayed, stores the time allocation 
editorial information as internal materials and converts time 
allocation editorial information stored in the internal 
material structure to the form that the outside can refer to. 

i;| The spatial allocation editorial information processing 

module 302 examines whether the spatial allocation editorial 

'■W information supplied from the outside is proper or not so as 
Cl to determine a spatial position of each media object existing 
s. in the contents, stores the spatial allocation editorial 

IfS information as internal materials and converts the spatial 
li| allocation editorial information stored in the internal 
material structure to the form that the outside can refer to. 

The user interactivity editorial information processing 
module 303 examines whether the user interactivity editorial 

20 information supplied from the outside is proper or not to edit 
the user interactivity so as to provide information what the 
users want in response to requests of users which are inputted 
in the form of a mouse click, a mouse touch, a keyboard input, 
etc., for each media object when the object-based interactive 

25 multimedia contents are displayed, stores the user 
interactivity editorial information as internal materials and 
converts the user interactivity editorial information stored 
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in the internal material structure to the form that the 
outside can refer to. 

The property and logical structure editorial information 
processing module 3 04 examines whether the editorial 
5 information supplied from the outside is proper or not so as 
to edit a logical structure such as a property determining a 
shape of each media object, a dependent relationship between 
media objects, grouping information of media objects, etc., 
stores the editorial information as internal materials and 
iS converts the editorial information stored in the internal 
^ material structure to the form that the outside can refer to. 
2jt Referring to Fig. 4, there is provided a detailed block 

diagram of the media coding/decoding unit 105 in Fig. 1 in 
*. accordance with an embodiment of the present invention. 
l|f A pre/post processor 401 performs a function for 

converting various types of audio/video data to an editable 
f** type and a function for automatically or semi-automatically 
dividing each body existing in the video and static images. 

A media coder 402 encodes media data such as video, audio 
20 and static images by using a standardized coding technique, 
e.g., MPEG, JPEG and so on, or a non-standardized coding 
technique to thereby produce a media stream. 

A media decoder 403 decodes a media stream encoded by the 
standardized coding technique or the non-standardized coding 
25 technique to thereby reproduce media data. 

Herein, the media coder 4 02 and the media decoder 403 
should be capable of encoding and decoding media objects of 
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the video and static images with an arbitrary shape, 
respectively, and they further contain a media processing 
accelerator in charge of performing a media coding/decoding 
function on a hardware accelerator in real-time or a higher 
5 speed than real-time in case there is hardware dedicated for 
the media processing in the system. 

Moreover, the media coder 4 02 includes more than one 
video coder and at least one audio coder, wherein the video 
coder encodes video data based on a standardized video coding 
10 technique or a non-standardized video coding technique to 
thereby generate a video stream and the audio coder encodes 
audio data based on a standardized audio coding technique or a 
W non-standardized audio coding technique to thereby produce an 
* audio stream. Like the media coder 4 02, the media decoder 403 
lip also contains more than one video decoder and at least one 
l|J audio decoder, wherein the video decoder decodes a coded video 
M stream which was encoded based on the standardized video 
coding technique or the non-standardized video coding 
technique to thereby reproduce video data and the audio 
2 0 decoder decodes a coded audio stream which was encoded based 
on the standardized audio coding technique or the non- 
standardized audio coding technique to thereby reconstruct 
audio data. 

As an example of the video coding /decoding technique, 
25 there is a coding/decoding technique following MPEG-4 video 
standard (ISO/IEC 14496-2), which is capable of coding and 
decoding video objects of an arbitrary shape. The audio coder 



15 



and decoder employ an audio coding/decoding technique 
satisfying MPEG-4 audio standard (ISO/IEC 14496-3). 

Fig. 5 represents a flow chart of showing an object- 
based interactive multimedia contents authoring method in 
5 accordance with an embodiment of the present invention. 

In step 501, it is determined whether a user authors new 
contents or edits pre-authored contents, i.e., whether current 
contents are new. 

As a result of step 501, if the current contents are 
1Q determined as the new contents, there is secured in step 502 
\9 an authoring space for authoring the new contents, i.e., a new 
© internal material structure and a new authoring space on a 
13 user interface. 

% Then, in step 5 03, the authoring space is initialized, 

lpj That is, various parameters to be used in securing the 
:■_ authoring space are inputted by the user or determined as 
| si preset defaults. 

Sequentially, this procedure goes to step 505 in which 
media objects required in the contents authoring and editing 
20 are inserted and unnecessary contents are deleted. 

As the result of step 501, if it is determined that the 
current contents are not the new contents, in step 5 04, 
contents in the binary file form or the text file form are 
read in when the pre-authored contents are opened and, then, 
25 this procedure proceeds to step 505. 

After inserting or deleting media objects required in 
editing/authoring the current contents in step 505, the 
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spatial allocation and time allocation information for each 
media object existing in the contents and properties 
characterizing each media object are edited in step 506 and 
the user interactivity is edited on the object basis in step 
5 507. 

In the above, a processing sequence of steps 505, 506 
and 507 can be changed depending on a selection of the user 
and those steps can be repeatedly performed according to a 
request of the user. 
Ill In step 50 8, it is checked whether the contents edited 

y'J during executing the authoring process are to be stored or not. 
19 As a result of step 508, if it is determined that the 

CI contents are not to be stored, this procedure goes to step 512 
8 : in which it is determined whether continuously performing the 
llij editing process or not. On the other hand, if it is decided 
U that the contents are to be stored, in step 509, it is checked 
L, whether this is a first storing process of the contents or not. 

As a result of step 509, if it is determined that the 
storing is the first time for the contents, a storing form of 
20 the contents is selected as the binary or text form in step 
510. If otherwise, the contents are to be stored in its 
original form in step 511. 

The editing process of steps 505 to 511 is repeatedly 
performed until it is determined in step 512 that the contents 
25 editing is completed. If the above editing and authoring 
process is completed, the editing procedure is terminated. 

The above methods in accordance with the present 
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invention can be implemented as computer programs which can be 
recorded on a computer readable recording medium such as CDROM, 
RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc. 

The present invention can obtain following advantages by 
5 providing an AV contents authoring apparatus and method 
capable of executing interactive manipulation on an object 
basis required in the object-based interactive multimedia 
contents authoring employed in integrated data broadcasting, 
interactive broadcasting, Internet broadcasting and so on. 
ItQ First, it is possible to carry out the user interactive 

■j\ editing capable of providing desired information on the media 
gl object basis in response to requests of users. 

PI Second, in authoring the object-based interactive 

multimedia contents, it is possible to execute the 
ifta editing/authoring process in which a body existing in a video, 
audio and static image is treated as one object, which is not 
supported by the conventional editing/authoring tools. 

Third, the video and static image object with an 
arbitrary shape can be used in the contents editing/authoring 
2 0 process. 

As a result, it is come true to develop the object-based 
multimedia contents editing/authoring technique and to 
accomplish the spread of various types of digital broadcasting 
services . 

25 While the present invention has been described with 

respect to the particular embodiments, it will be apparent to 
those skilled in the art that various changes and 
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modifications may be made without departing from the spirit 
and scope of the invention as defined in the following claims. 
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